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Abstract — It is known that state-dependent, multi-step Lya- 
punov bounds lead to greatly simplified verification theorems 
for stability for large classes of Markov chain models. This is 
one component of the "fluid model" approach to stability of 
stochastic networks. In this paper we extend the general theory 
to randomized multi-step Lyapunov theory to obtain criteria 
for stability and steady-state performance bounds, such as finite 
moments. 

These results are applied to a remote stabilization problem, 
in which a controller receives measurements from an erasure 
channel with limited capacity. Based on the general results in 
the paper it is shown that stability of the closed loop system 
is assured provided that the channel capacity is greater than 
the logarithm of the unstable eigenvalue, plus an additional 
correction term. The existence of a finite second moment in 
steady-state is established under additional conditions. 



I. Introduction 

Stochastic stability of Markov chains has a rich and com- 
plete theory, and forms a foundation for several other general 
techniques such as dynamic programing and Markov Chain 
Monte-Carlo (MCMC) [26]. This paper concerns extensions 
and application of a class of Lyapunov techniques, known as 
state-dependent drift criteria [29]. This technique is the basis 
of the fluid-model (or ODE) approach to stability in stochastic 
networks and other general models [2], [12], [23], [15], [9], 
[10], [26], [13]. 

In this paper we consider a stability criterion based on a 
state-dependent random sampling of the Markov chain of the 
following form: It is assumed that there is a function V on the 
state space taking positive values, and an increasing sequence 
of stopping times {71 : i € N+}, with 7o = 0, such that for 
each i, 

E[V(x Ti+1 ) I JV,] < V(x % ) - S(x Ti ) (1) 

where the function S : X — > R is positive (bounded away 
from zero) outside of a "small set", and Fji denotes the 
filtration of "events up to time %"■ Under suitable conditions 
on the Markov chain, the drift 5, and the sequence {%}, we 
establish corresponding stability and ergodicity properties of 
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the chain. The main results extend and unify previous research 
on stability and criteria for finite moments obtained in [23], 
[29], [15], [10], [26]. 

Motivation for this research arose from our interest in 
applications to networked control, and information theory with 
variable length and variable delay decoding [5], [35], [34], 
and non-asymptotic information theory [33]. Specifically, in 
some network protocol, team decision and networked control 
applications there is only intermittent access to sensor infor- 
mation, control action or some common knowledge on which 
decisions can be made. The timing may be random, depending 
on availability of communication resources. One example of 
such conditions is reported in [41], for establishing stochastic 
stability of adaptive quantizers for Markov sources where 
random stopping times are the instances when the encoder 
can transmit granular information to a controller. We will 
also consider such an application in detail in the paper. In 
this context, there has been a significant amount of research 
on stochastic stabilization of networked control systems un- 
der information constraints. For a detailed review see [42]. 
Stochastic stability of adaptive quantizers have been studied 
both in the information theory community (see [16], [20]) as 
well as control community ([4], [22], [32], [41]). [32] provided 
a stability result under the assumption that a quantizer is 
variable-rate for systems driven by noise with unbounded 
support for its probability measure. [32] used asymptotic 
quantization theory to obtain a time-varying scheme, where 
the quantizer is used at certain intervals at a very high rate, 
and at other time stages, the quantizer is not used. For such 
linear systems driven by unbounded noise, [41] established 
ergodicity, under fixed-rate constraints, through martingale 
methods. These papers motivated us to develop a more general 
theory for both random-time drift as well as the consideration 
of more general noise models on the channels. In a similar 
line of work, [17] also considered stability of the state and 
quantization parameters, [30] studied the problem concerning 
time-varying channels and provided a necessity and sufficiency 
result for boundedness of second moments, [40] studied the 
problem of control over an erasure channel in the absence 
of noise, and [44] considered discrete noisy channels with 
noiseless feedback for systems driven by Gaussian noise using 
the machinery developed in the current paper. [42] obtained 
conditions for the existence of an invariant probability measure 
for noisy channels, considering deterministic, state-dependent 
drift, based on the criteria developed in [29]. 

We believe our results will provide constructive tools to 
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address related stability issues in a large class of networked 
control problems. 

The contributions of this paper can be summarized as 
follows 

> Stochastic stability theory for Markov chains based on 
random-time, state-dependent stochastic drift criteria. A 
range of conditions are used to establish conditions for 
positive recurrence, and existence of finite moments. 

> The results are applied to stochastic stabilization over 
an erasure network, where a linear system driven by 
Gaussian noise is controlled. This paper establishes that 
in such an application of stabilization of an unstable 
system driven by Gaussian noise over erasure channels, 
for stochastic stability, Shannon capacity is sufficient 
(up to an additional correction term). For the existence 
of finite moments, however, more stringent criteria are 
needed. Regarding information rate requirements, our 
construction is tight up to an additional symbol, in 
comparison with necessary conditions presented in [30]. 

The remainder of the paper is organized as follows. The 
implications of the drift criterion (Q]i to various forms of 
stochastic stability are presented in the next section. The 
rest of the paper focuses on an application to control over 
a lossy erasure network with quantized observations. The 
paper ends with concluding remarks in Section [IV] Proofs of 
the stochastic stability results and other technical results are 
contained in the appendix. 

II. Stochastic Stability 

A. Preliminaries 

We let <fi — {<fit,t > 0} denote a Markov chain with state 
space X. The basic assumptions of [27] are adopted: It is 
assumed that X is a complete separable metric space, that 
is locally compact; its Borel er-field is denoted £>(X). The 
transition probability is denoted by P, so that for any <fi E X, 
A E £>(X), the probability of moving in one step from the state 
4> to the set A is given by P(0t+i <E A \ <p t = 4>) = P(4>, A). 
The n-step transitions are obtained via composition in the 
usual way, P ((/>*+„ G A | <p t = 4>) = P n {4>, A), for any n > 1. 
The transition law acts on measurable functions / : X — > K and 
measures fi on B(X) via, 

Pf ((/>):= [ P(^dy)f(y), cj> G X, 
Jx 

and 

fiP(A):= //i(#)P(^A), AgB(X). 
Jx 

A probability measure ir on B(X) is called invariant if irP = 
ir. That is, 

J n(d(j))P(cj), A) = tt(A), A G B(X). 

For any initial probability measure v on B(X) we can 
construct a stochastic process with transition law P, and 
satisfying tfio ~ v. We let P„ denote the resulting probability 
measure on sample space, with the usual convention for v = 
8$ when the initial state is <f) G X. When v = tt then the 
resulting process is stationary. 



There is at most one stationary solution under the following 
irreducibility assumption. For a set A G BiX) we denote, 

t a := min(t > 1 : (j) t G A) (2) 

Definition 2.1: Let ip denote a sigma-finite measure on 
B(X). 

(i) The Markov chain is called (p-irreducible if for any 
4> G X, and any B G B(X) satisfying ip(B) > 0, we 
have 

P {t b < oo} > . 

(ii) A (^-irreducible Markov chain is aperiodic if for any 
G X, and any B G B(X) satisfying tp(B) > 0, there 
exists no = no((j>, B) such that 

P n {(j),B) > for all n > n Q . 

(iii) A (,5-irreducible Markov chain is Harris recurrent if 
^4>{tb < oo) = 1 for any 4> E X, and any B E B(X) 
satisfying tp(B) > 0. It is positive Harris recurrent if in 
addition there is an invariant probability measure 7r. 

Intimately tied to (^-irreducibility is the existence of a suit- 
ably rich collection of "small sets", which allows Nummelin's 
splitting technique to be applied leading to verification for 
Harris recurrence. A set A G B(X) is small if there is an 
integer no > 1 and a positive measure /i satisfying /i(X) > 
and 

P no (x,B) > fi(B), for all x E A, andB G B(X). 

Small sets are analogous to compact sets in the stability theory 
for (^-irreducible Markov chains. In most applications of ip- 
irreducible Markov chains we find that any compact set is 
small - In this case, cf> is called a T-chain [27]. 

To relax the (/3-irreducibility assumption we can impose 
instead the following continuity assumption: A Markov chain 
is (weak) Feller if the function Pf is continuous on X, for 
every continuous and bounded function / : X — >• R. 

We next introduce criteria for positive Harris recurrence for 
</>irreducible Markov chains, and criteria for the existence of 
a steady-state distribution tt for a Markov chain satisfying the 
Feller property. 

B. Drift criteria for positivity 

We now consider specific formulations of the random- 
time drift criterion (Q]). Throughout the paper the sequence 
of stopping times {% : i E N+} is assumed to be non- 
decreasing, with 7o = 0. In prior work on state-dependent 
criteria for stability it is assumed that the stopping times take 
the following form, 

7I+i =77 + ra(<£(77)), t>0 

where n: X — > N is a deterministic function of the state. 
The results that follow generalize state dependent drift results 
in [29] to this random-time setting. We note that a similar 
approach has been presented recently in the literature in [14] 
for random-time drift (see Theorem 4), which readily gener- 
alizes the state dependent drift results in [29]. The conditions 
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presented in [14] are more restrictive for the stopping times 
than what we present here. Furthermore we present discussions 
for existence of finite moments, as well as extensions for non- 
irreducible chains. 

The proofs of these results are presented in the appendix. 

Theorem 12.11 is the main general result of the paper, pro- 
viding a single criterion for positive Harris recurrence, as well 
as finite "moments" (the steady-state mean of the function / 
appearing in the drift condition (O). The drift condition © is 
a refinement of (Q~|i. 

Theorem 2.1: Suppose that <p is a (^-irreducible and aperi- 
odic Markov chain. Suppose moreover that there are functions 
V: X -> (0,oo), 5: X -> [l,oo), /: X [l,oo), a small set 
C, and a constant b £ R, such that the following hold: 

E[^(0r 2+1 ) I JrJ < V^rJ ~ 5{<t>%) + W^gc} 



r z+ i-i 

k=% 



/(&) I Jr. <5(<h-,) 



z>0. 



(3) 



Then the following hold: 

(i) is positive Harris recurrent, with unique invariant 
distribution 7r 

(ii) tt(/) := / f(cj>) tt(#) < oo 

(iii) For any function g that is bounded by /, in the sense 
that supj, |<?(<^)|//(<^) < oo, we have convergence of 
moments in the mean, and the Law of Large Numbers 
holds: 

MM 



lim Eri 



1 



N-l 



lim — , 

N->oo N ^— ' 
t=0 



9{<i>t) 



a.s. 



G X 



□ 



Remark 2.1: We note that, for (ii) in Theorem 12.11 the 
condition that / : X — > [1, oo), 6: X — > [1, oo), can be relaxed 
to /: X — > [0, oo), 5: X — > [0,oo), provided that one can 
establish (i), that is the positive Harris recurrence of the chain 
first. 

This result has a corresponding, albeit weaker, statement for 
a Markov chain that is Feller, but not necessarily satisfying the 
irreducibility assumptions: 

Theorem 2.2: Suppose that is a Feller Markov chain, 
not necessarily (^-irreducible. If in addition (O holds with C 
compact, then there exists at least one invariant probability 
measure. Moreover, there exists c < oo such that, under any 
invariant probability measure tt, 



= / 7T(d0)/fo) < C. 



(4) 



□ 



We conclude by stating a simple corollary to Theorem 12. II 
obtained by taking f(<j)) = 1 for all e X. 

Corollary 2.1: Suppose that ^ is a (^-irreducible Markov 
chain. Suppose moreover that there is a function V : X — » 
(0,oo), a small set C, and a constant b € R, such that the 
following hold: 

E[V^t; +1 ) I J r J < V(0r.) - 1 + w {^ec } 
sup E[T z+ i - T z | Jr.] < oo- (5) 

z>0 



Then is positive Harris recurrent. 



III. 



□ 



Application to Stochastic Stabilization over 
an Erasure Channel 

The results of the previous section are now applied to a 
remote stabilization problem, in which the plant is open-loop 
unstable, and the controller has access to measurements from 
an erasure channel — see Figure Q] 



Coder 



Channel 



Q 


Controller 







Plant 



Fig. 1: Control over a discrete erasure channel with feedback. 
Coder represents the quantizer and the encoder. 

We begin with a scalar model; extensions to the multivariate 
setting are contained in Section ITlI-EI 

A. Scalar control/communication model 

Consider a scalar LTI discrete-time system described by 

Xt+i = ax t + bu t + d t , t > (6) 

Here Xt is the state at time t, u t is the control input, xq is a 
given initial condition, and {d t } is a sequence of zero-mean 
independent, identically distributed (i.i.d.) Gaussian random 
variables. It is assumed that \a\ > 1 and 6^0: The system is 
open-loop unstable, but it is stabilizable. 

This system is connected over an erasure channel with finite 
capacity to a controller, as shown in Figure Q] The controller 
has access to the information it has received through the 
channel. The controller estimates the state and then applies 
its control. We will establish bounds on data rates which lead 
to various versions of stochastic stability for the closed loop 
system. 

The details of the communication channel are specified as 
follows: The channel source consists of state values, taking 
values in R. The source is quantized: The quantizer, at time 
t > 0, is represented by a map Q t : R — >• R, characterized by 
a sequence of non-overlapping intervals Vt '■= {Bi,t}, with 
\P\ = K + 1, such that Qt(x) — q\ if and only if x 6 B,^; 
that is, 

Qt(x) = ^2qt X l{ x gs M }' 

i 

The quantizer outputs are transmitted through a memoryless 
erasure channel, after being subjected to a bijective mapping, 
which is performed by the channel encoder: The channel 
encoder maps the quantizer output symbols to corresponding 
channel inputs q € M. := {1, 2 . . . , K + 1}. An encoder at 
time t, denoted by £ t , maps the quantizer outputs to A4 such 
that S t (Qt(x t )) =q t eM. 

The controller/decoder has access to noisy versions of the 
coder outputs for each time, which we denote by {q 1 } G 
M U {e}, with e denoting the erasure symbol, generated 
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according to a probability distribution for every fixed q G M.. 
The channel transition probabilities are given by: 



P{q 



i\q 



: P, 



P(q> 



e\q 



i G M. 



For each time t > 0, the controller/decoder applies a 
mapping V t : MU {e} — > R, given by: 

2>t&) = X hli^e} + X \ {q , =e} 

We restrict the analysis to a class of uniform quantizers, 
defined by two parameters: bin size A > 0, and an even 
number K > 2. The set A4 consists of K + 1 elements. 
The uniform quantizer map is defined as follows: For k = 
1,2..., K, 



(fc-±(tf + l))A, 

if x G [(k - 1 - §.fQA, (fc - §iT)A) 



(i(iC-l))A, if x 



§#A 



0. 



if xg [-\KA lifA]. 



We consider quantizers that are adaptive, so that the bin 
process can vary with time. The bin size A t at time t is 
assumed to be a function of the previous value A t _i and the 
past channel output q' t _ 1 - 

B. Stochastic stabilization over an erasure channel 

Consider the following time-invariant model. Let T t denote 
an i.i.d. binary sequence of random variables, representing 
the erasure process in the channel, where the event T t = 1 
indicates that the signal is transmitted with no error through the 
erasure channel at time t. Let p = E[T f ] denote the probability 
of success in transmission. 

The following key assumptions are imposed throughout this 
section: Given K > 2 introduced in the definition of the 
quantizer, define the rate variables 



R:=\og 2 (K + l) R' = \og 2 (K) 
We fix positive scalars 5, a satisfying a < 1, and 



and 



a > \a\2 



a(\a\ +6)P~ 



-R' 



< 1. 



(7) 



(8) 



(9) 



We note that the (Shannon) capacity of such an erasure 
channel is given by log 2 (-ftT + l)p [8]. From ©-(O it follows 
that if \og 2 (K)p > log 2 (|a|), then a, S exist such that the 
above are satisfied. 

To define the bin-size update rule we require another 
constant L > 0, chosen so that L 1 =; aL > 1, where we take 
the lower bound as 1 for convenience; any positive number 
would suffice. 

Define the mapping H : R x R x {0, 1} ->• M, 



H(A,h,p) = \a\+S 
H{A,h,p) = a 
H(A,h,p) = l 



if \h\>l, or p = 
if < \h\ < l,p= 1,A > L 
if < \h\ < l,p= 1,A < L 



Then with Aq > L selected otherwise arbitrarily, define 



u, 



a , 

- b x t , 



A t+1 = A t H(A u \ht\,r t ),where h t 

The update equations above imply that 
A t > La =: L' > 1 . 



■ri 



A t 2* 



7^(10) 



(11) 



Given the channel output q' t ^ e, the controller can deduce 
the realization of T t and the event {\h t \ > 1} simultaneously. 
This is due to the observation that if the channel output is 
not the erasure symbol, the controller knows that the signal is 
received with no error. If q' t = e, then the controller applies 
as its control input and enlarges the bin size of the quantizer. 

Lemma 3.1: Under ( flOl ). the process (x t ,A t ) is a Markov 
chain. 

Proof: The system state dynamics can be expressed Xt+i = 
axt — axt + dt, where i t = T t Q^*(xt). It follows that the 
pair process (x t , A t ) evolves as a nonlinear state space model, 



x t+1 = a(x t - T t Q K t {x t )) + d t 



A t+ i = A t H(A t , 



x, 



,T t ) 



(12) 



'2«'-iA t 

in which (d t , T t ) is i.i.d.. Thus, (x t , A t ) form a Markov chain 
(see [27, Ch. 2]). □ 

Our result on the existence and uniqueness of an invariant 
probability measure is the following. 

Theorem 3.1: For an adaptive quantizer satisfying ©-Q, 
suppose that the quantizer bin sizes are such that their base- 
2 logarithms are integer multiples of some scalar s, and 
\og 2 (H( ■ )) takes values in integer multiples of s. Then the 
process (xt,A t ) forms a positive Harris recurrent Markov 
chain, with a unique invariant probability measure it. If the 
integers taken are relatively prime (that is they share no 
common divisors except for 1), then the invariant probability 
measure is independent of the value of the integer multiplying 

s. a 

Under slightly stronger conditions we obtain a finite second 
moment: 

Theorem 3.2: Suppose that the assumptions of Theo- 
rem 13.11 hold, and in addition we have the bound 

P 



a'll-p 



< 1. 



(2 R - l) 2 . 

It then follows that for each initial condition (xq, Ao), 
lim E[a; 2 ] = E^x 2 ,] < oo . 

t— >oo 



(13) 



□ 



Remark 3.1: We note that Minero et al [30], in Theorem 
4.1, observed that a necessary condition for mean square 
stability is that the following holds: 

P 



\a\ 2 (l-p 



(2*)* 



< 1. 



Thus, our sufficiency proof almost meets this bound except 
for an additional transmitted symbol. 
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We now consider the mth moment case. This moment can 
become useful for studying multi-dimensional systems for a 
sequential analysis of the modes, as we briefly discuss in 
Section HTFE1 

Theorem 3.3: Consider the scalar system in @. Let m <E 
N, suppose that the assumptions of Theorem 13.11 hold, and in 
addition we have the inequality, 

\a\ m ( 1-P+ -, 5 P r ) < 1. 

1 1 V 1 (2 R - l) m J 

It then follows that with the adaptive quantization policy 
considered, lim^oo E[|x t | m ] = [|a;o | m ] < oo . □ 

C. Connections with the drift criteria and the proof program 

Stability of the control/communication model is established 
using the random-time stochastic drift criteria presented in the 
previous section, applied to the Markov chain tj> = (xt,A t ) 
(see Lemma [3~TT i. We provide an overview here, and the details 
can be found in the appendix. 




Id = F 



Fig. 2: Drift in the Markov Process. When under-zoomed, the 
error increases on average and the quantizer zooms out; when 
perfectly-zoomed, the error decreases and the quantizer zooms 
in. 

Figure [2] provides some intuition on the construction of 
stopping times and the Lyapunov functions. Recall that ht — 
Xt/(2 R _1 At) was introduced in ( fTob . The arrows shown in 
the figure denote the mean one-step increments of (xt,ht): 
That is, the arrow v with base at (x, h) is defined by, 

v = £[{x t+ i,h t+ i) - (x t ,h t ) | (x t , h t ) = (x, h)] 

With F > fixed, and with F' = F2- ( ~ R '~ 1 \ two sets 
are used to define the small set in the drift criteria, C x = 
{x : \x\ < F} for F > 0, and C\ = {A : A < F'}. 
Denote Ch = {h : \h\ < 1}, and assume that F > is 
chosen sufficiently large so that (xt, A t ) S C x x Ca whenever 
(xt, ht) € C x x Ch- When Xt is outside C x and h t outside Ch 
(the under-zoomed phase of the quantizer), there is a drift for 
h t towards Ch- When the process xt reaches Ch (the perfectly- 
zoomed phase of the quantizer), then the process drifts towards 
C x . 

We next construct the sequence of stopping times required 
in the drift criteria of Section III-BI The controller can receive 
meaningful information regarding the state of the system when 



two events occur concurrently: the channel carries information 
with no error, and the source lies in the granular region of the 
quantizer: That is, x t G [-\KA t , \K A t ] (or \h t \ < 1) and 
T f = 1. The stopping times are taken to be the times at which 
both of these events occur. With, \ho\ < 1, To = 1, we define 
% = and 

T z +i = wf{k>T z :\h k \<l,? k = l}, zeN. 

These stopping times are nearly geometric when the bin 
size is large. The proof of Proposition 13.11 is presented in 
Section EED 

Proposition 3.1: The discrete probability measure 

P(T z +i — T z — k | xt z , A7;) satisfies, 

{l-pf- 1 < P(T Z+1 -T Z > k\x % ,A % ) < (l-pf-i+oQ.), 
where o(l) — > as A7; — > 00 uniformly in x^ z - □ 

The next step is to establish irreducibility structure. The 
proof of the following is contained in Section IV-B.2I 

Proposition 3.2: Under the assumptions of Theorem 13.11 
the chain (xt,A t ) is (,5-irreducible for some tp, it is aperiodic, 
and all compact sets are small. □ 

We now provide sketches of the proofs of the main results. 
The details are collected together in the appendix. 

Sketch of Proof of Theorem \3.1\ The logarithmic function 
Vo(xt, At) = log(A 2 ) + Bo for some Bo > serves as the 
Lyapunov function in (f3]), with f(x, A) set as a constant. Note 
that by CD), Vb(a; t ,A t ) > 0. 

Together with Propositions 13.11 and 13.21 we apply Theo- 
rem 12.11 in the special case of Corollary 12.11 Proposition 13.21 
implies the existence of a unique invariant measure. Details of 
the proof are presented in section IV-B.3I □ 

Sketch of Proof of Theorem \3.2i A quadratic Lyapunov 
function V2(x t ,A t ) = A 2 is used, along with 5(xt,A t ) — 
eAf for some e > 0, and f(x t , A t ) = £x% for some £ > 0. 
The bound (f3]) is established in Section [V-B.4l so that the limit 
lim^oo E[xj] exists and is finite by Theorem 12.11 Details of 
the proof are presented in section IV-B.4I □ 

Sketch of Proof of Theorem \3.3i Theorem 12.11 is applied 
with the Lyapunov function V m (xt, A t ) = A™, S(xt,A t ) = 
eA™ for some e > 0, and f(x t , At) = t,\x t \ m for some £ > 0. 
Details of the proof are presented in section IV-B.5I □ 

D. Simulation 

Consider a linear system 

x t+ i = ax t +u t + d t , 

with a = 2.5, {d t } is an i.i.d. N(Q, 1) Gaussian sequence. 
The erasure channel has erasure probability 1 — p = 0.1. For 
stability with a finite second moment, we employ a quantizer 
with rate 

kgatiy ^,^ J ^l + 1) = log a (5) 

bits. That is, a uniform quantizer with 5 bins. We have taken 
L' = 1. Figures [3] and [4] illustrate the conclusions of the 
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stochastic stability results presented in Theorems 13.11 and 
13.21 The plots show the under-zoomed and perfectly-zoomed 
phases, with the peaks in the plots showing the under-zoom 
phases. For the plot with 5 levels, the system is positive Harris 
recurrent, since the update equations are such that a — 0.629, 
S = 0.025 and log 2 (#(•)) € {-0.6744,0,1.363}. These 
values satisfy the irreducibility condition since —0.6744 = 
— 1.363/2, and hence the communication conditions in Theo- 
rem [XT] are satisfied. Furthermore, (O is satisfied since 



*(\a\+6) p ~ 



< 0.698 < 1. 



Increasing the bit rate by only two bits in Figure [4] leads to 
a much more desirable sample path, for which the severity of 
rare events is reduced. 
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Fig. 3: Sample path for a stochastically stable system with a 
5 -bin quantizer. 
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Fig. 4: Sample path with a 17-bin quantizer; a more desirable 
path. 



E. Extension to multi-dimensional systems 

The control laws and analysis can be extended to multi- 
dimensional models. Consider the multi-dimensional linear 
system 



x t+1 = Ax t + Bu t + d t 



(14) 



where xt <E W 1 is the state at time t, ut is the control 
input, and {d t } is a sequence of zero-mean independent, 
identically distributed (i.i.d.) R™-valued zero-mean Gaussian 
random variables. Here A is the system matrix with at least one 
eigenvalue greater than 1 in magnitude, so that the system is 
open-loop unstable. Without any loss of generality, we assume 
A to be in Jordan form. Suppose that B is invertible for ease 
in presentation. 

The approach for the scalar systems is applicable, however 
some extension is needed. Toward this goal, one can adopt 
two approaches. 

In one scheme, one could consider a sequential stabilization 
of the scalar components. In particular, we can perform an 
analysis by considering a lower to upper sequential approach, 
considering stabilized modes at particular stopping times as 
noise with a finite moment. Using an inductive argument, one 
can first start with the lowest mode (in the matrix diagonal) 
of the system, and stabilize that mode so that there is a finite 
invariant mth moment of the state. We note that the random 
process for the upper mode might not have Markov dynamics 
for its marginal, but the joint system consisting of all modes 
and quantizer parameters will be Markov. Such a sequential 
scheme ensures a successful application of the scalar analysis 
presented in this paper to the vector case. One technicality 
that arises in this case is the fact that the effective disturbance 
affecting the stochastic evolution of a repeated mode in a 
Jordan block is no longer Gaussian, but can be guaranteed 
to have a sufficiently light tail distribution by Theorem 13.31 

Another approach is to adopt the discussions in [45] and 
Section IV of [44] (see also [25] for related constructions) 
and apply a vector quantizer by transmitting the quantizer 
bits for the entire M n -valued state. In particular, by defining a 
vector quantizer as a product of scalar quantizers along each 
(possibly generalized) eigenvector with a common under-zoom 
bin, letting h 1 denote the ratio of the state and the bin range of 
the corresponding ith scalar quantizer and defining a sequence 
of stopping times as follows with 7o = and for z £ N 



Tm+i = inf{fc > T z ■ \K\ <1.* 



1,2. 



..,71, 



1}, 



the analysis can be carried over through a geometric bound 
on the distribution of subsequent stopping times, obtained by 
an application of the union bound. See [44] for details. 

IV. Concluding Remarks 

This paper contains two main contributions. One is on 
a general drift approach for verifying stochastic stability of 
Markov Chains. The other is on stabilization over erasure 
channels. We believe that the results presented in this paper 
will have many applications within the context of network 
stability, and networked control systems as well as information 
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theoretic applications. Important previous research on per- 
formance bounds for variable-length decoding schemes use 
stopping time arguments [5], [33], and this could form the 
starting point of a Lyapunov analysis. 

The methods of this paper can be extended to a large class 
of networked control systems and delay-sensitive information 
transmission. For networked control systems, the effects of 
randomness in delay for transmission of sensor or controller 
signals (see for example [31], [6]) is an application area where 
the research reported in this paper can be applied. 

Rates of convergence under random-time drift is one direc- 
tion of future research. It is apparent that the nature of the drift 
as well as the distribution of stopping times used for drift will 
play a role in the rate of convergence. We refer the reader to 
[11] and [7], for results when the drift times are deterministic. 

Positive Harris recurrence can be a crude measure of sta- 
bility, as seen in the numerical results in this paper. Com- 
puting the sensitivity of performance to the bit rate is an 
important future research problem for practical applications. 
For example, it was observed in Figure |4] that increasing the 
bit rate by only two bits leads to much more desirable sample 
paths, and the magnitudes of rare events are significantly 
reduced. A Markovian framework is valuable for sensitivity 
analysis, as applied in the reinforcement learning literature 
(see commentary in Section 17.7 of [27]). 

V. Appendix 
A. Proofs of the Stochastic Stability Theorems 



1 ) Proof of Theorem 12.71 ( i): The proof is similar to the 
proof of the Comparison Theorem of [27]: Define the sequence 

{M z : z > 0} by M = V(0o). and for z > 0, 

Z 

fc=0 

Under the assumed drift condition we have, 

z-l 

E[M Z+1 | JVJ < 7(W+J](%J - W { ^ eC} ), 

fc=0 

which implies the super-martingale bound, 

E[M Z+1 | 7VJ < M z 

For a measurable subset C C X we denote the first hitting 
time for the sampled chain, 



Cc = min{z > 1 : cj) Tz e C} 



(15) 



Define £g = min(n, Cc) for any n > 1. Then E[Mj»] < Mq 
for any n E Z, and 



CS-i 



E[E ^rJl^o] <M + b. 



k=0 



Applying the bound E^^Lr* 1 f(0k) I Ft z < 5(4>r z ) and 
that f{4>) > 1, the following bound is obtained from the 



smoothing property of the conditional expectation: 

.CS-i 



E[T CS | ^o] = e[£ E[T i+ x-n\T Q ] 

< Mo + b 



i=0 
Cc-1 

<E[]T 5(^)1-^0 



i=0 

Hence by the monotone convergence theorem, 

E[ro] < E[T Cc ] = lim E[T CS | T Q ] < M + b. 
Consequently we obtain that 

sup E[tc] < oo, 

as well as recurrence of the chain, P^(rc < oo) = 1 for any 
4> £ X. Positive Harris recurrence now follows from [28] Thm. 
4.1. □ 

The following result is key to obtaining moment bounds. 
The inequality (Tl7] > is known as drift condition (V3) [27]. 
Define, 

TC — 1 



V f *(4) 



t=0 



e x. 



(16) 



Lemma 5.1: Suppose that <j> satisfies all of the assumptions 
of Theorem 12.11 except that the ?/>-irreducibility assumption is 
relaxed. Then, there is a constant bf such that the following 
bounds hold 

pv; < vf-f + bfic (i7) 

Vf(4>) < V(cf>) + b f , 0eX. (18) 
Proof: The drift condition (fTTT i is given in Theorem 14.0.1 
of [27]. 

The proof of ( fT8l is based on familiar super-martingale 
arguments: Denote M = V^(<t5o)> and and for z > 0, 

T.+1-1 

M z+1 = V(cl> Tx+1 )- J2 (-f(M + bl { ^ Tke c}) (19) 

k=0 

The super-martingale property for {M z } follows from the 
assumed drift condition: 



E[M 2+1 | T T .\ = M z 



V(<(> %+1 )-V(<j> % ) 



< MM 



k=T z 



As in the previous proof we bound expectations involving 
the stopping time Cc beginning with its truncation = 

min(n,Cc)- 

The super-martingale property gives E[Mq»] < M Q , and 
once again it follows again by the monotone convergence 
theorem that Vf satisfies the bound (fT8l as claimed. □ 



s 



2) Proof of Theorem 12.71 (ii) and (Hi): The existence of Without any loss, let z = 0, 7o = 0, so that 0^, = P xo Ao (71 > 
a finite moment follows from Lemma [5TTI and the following fc). 

generalization of Kac's Theorem (see [27, Theorem 10.4.91): NT . .. n • • f „ tU 

& J/ Now, at time 0, upon receiving a message successfully, the 

/r /t a -i v estimation error satisfies |xq — xq\ < Ao/2, as such we have 

7t(#)/(<£) = y tt(#)EJ £ /(&)J, (21) that |o||x + (6/o)«o| < k|A /2. 

The probability 0fc for fc > 2 is bounded as follows: 

where A is any set satisfying tt(A) > 0, and ta — inf(i > 
1 : (fit € A), The super-martingale argument above ensures 
that the expectation under the invariant probability measure is 

bounded by recognizing C as a recurrent set. Qfc = f p| ( Ts = ) y (|^| > x) \ 

(iii) now follows from the ergodic theorem for Markov \ '_' / 

chains, see [27, Theorem 17.0.1]. □ ,fe-i 

3) Proof of Theorem |22} The existence of an invariant - Px °' A <> ( D ( Ts = °) U - 2 ^ ^"l + S ^ 1( * A o) J 
probability measure in (i) follows from Theorem 12.3.4 of f _1 

/ K, — 1 

[27] (the solution to the drift condition (V2) can be taken _ p [ O (T — 0) 

to be the mean hitting time, V((j>) = E [r c ]). See also [21, ~ z ,A ^| |l «, - ) 
Theorem 3.1]. s _ 1 

The proof of (ii) is similar. Rather than work with the mean U(|a s (a;o + ^ a~* -1 di)| > (|a| + «S) a_1 2 fl ' -1 Q!Ao) 

return time to C, we consider the function Vj defined in i=0 
Lemma IBTT] We have by the Comparison Theorem of [27], ,fe-2 

n-l 

< P n Vf < Vf + nb f - P t S 

t=o 

Hence for any <fi € X, 

n-l 



< P.o.Aol r\(?. = 0)U(\h a \>l) T fc _i=0)(l-p) 

^ s=l 
, k-2 

+P XoAo ( f|(T s = 0)U(|^|>l) 

^ s=l 

lim sup - '^T pt f OA) < b f- ( 22 ) \\a k - 1 (x + (b/a)uo + ^r f a,- i - 1 d i 



k-2 



i=0 
\fe-2 -R'-l, 



Suppose that ir is any invariant probability measure. Fix N < > (|a| + 6) 2 aAo 

oo, let /a? = min(7V, /), and apply Fatou's Lemma as follows, 



,1^ x xP^(\a h - l (x Q + (b/a)uo + Y2 a ~ i ~ ld i)\ 

tt(/jv) = lim sup tt( - > P7iV ) V i=0 



k-2 



t=0 



l"-x > (lol+O^^aAo) (24) 

< Trflim sup - ^ P*/jv) < &/• 



n— foo Tl £_q * /k2 



< p,o,ao n (TT, = 0) U > 1) | *T fc _ x = ) (1 - p) 



s=l 



Fatou's Lemma is justified to obtain the first inequality, 

because /jy is bounded. The second inequality holds by (l22l k-2 
and since f N < f. The monotone convergence theorem then + P^o.Ao ( |a fc_1 (xo + (b/a)u + ^ o _i_1 di 

gives 7r(/) < &/. □ 1 



i=0 
\fe-2 fl'-l. 



> (|o| + S)«-*2 H "^Ao 



fe-2 



^(H+Jf-^^aAo . (25) 



B. Proofs of Stability: Stochastic Stabilization over an Erasure _ p A (71 > fc — 1)(1 — p) 
Channel 

1) Proof of Proposition 13.71 We obtain upper and lower + P Ko ,a ( |a fe 1 (ceo + (b/a)uo + '^^a~ l ~ 1 d 

bounds below. ^ i=o 

Lemma 5.2: The discrete probability measure P(71+i — 
71 = fc | x f z , A 7; ) satisfies 

P(71+i -% > k\x Tz ,A r J < (1 -pf- 1 +G fc (A r J, 

where Gk(^r^) ^ > as A7; — > 00 uniformly in X7;. □ In the above derivation, (l24i > follows from the following: For 

_ , any three events M, C, D in a common probability space 

Proof: Denote for fc e N, 

e fc := P(71+i -%>k\x Tz , A r J 

= Px r ^A r ,(r z +i - 71 > fc). (23) P(A7n(CU73)) = P((MnC)U(A7n£>)) < P(MnC)+P(MnD) 
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Now, observe that for k > 2, 



fc-2 



Pxo,A ( K^o + (b/a)u + 1 1 d i 



i=0 



> (M±V 2 2*'-^A 
a a 



fe-2 



< 2P, ,a ^a-^di 



i=0 



> (2 ^-i ( 4 + i )fc - 2 « _l )Ao 



a 2' 



< 



C exp (- 



2a' 2 



)■ 



(26) 



where d26l l follows from ([8), for this condition ensures that 
the term 

(^^) fe - 2 £-~) 
|a| |a| 2 

is positive for k > 2, and bounding the complementary error 
function by the following: fj,[dx) < q^ 1 x/j,(dx), for 
q > 0. In the above derivation, the constants are: 



,2 _ E[d\] t _ H + 8 



I - a 



2 R ' 



(\a\)l* 



and 



C = 2<r' 



1 



27r(2iV - l)A /2 



Let us define: 



and 



((g fc - 2 jV-l/2)A ) 2 
2ct' 2 

((g fc jV-l/2)A ) 2 
2a' 2 



We can bound the probability 6^ defined in (1231 1. Since a 
decaying exponential decays faster than any decaying polyno- 
mial, for each m E N + , there exists an M < oo such that for 
aU k € N, 



Ce~ ak < MS: 



(27) 



Thus, we have that 



fe-2 



P,o,Ao(^0 + E a ^ > (^^ 22 ^ 1 T^ A °) Px ,Ao( n(T. = 0)U(|A.| > 1)) > P, .A(n' T - »' 



where 



fc-i 



G fc (A ro ):=^M(l-p) fc - s - 1 H- 



(31) 



s=l 



It now follows that, 

fe-i 

G fe (A To ) = ^M(l-p) fc - s - 1 H s - 



s=l 



fe-1 



A - 2m ^M(l-p) s 



fc - s -i/(fiV-l/2) ; 



fc-i 



= A - 2m (l-p) fe - 1 ^M(l-p) 



s=l 



(2 CT ' 2 ) 

(C^- 1/2) 2 X ~ m 

(2^r 



A 2m (l-p) fc - 1 
fc-i 

x M(l - p)- s r 2ms (N - _)- 2 ™(2a' 2 r 



s=l 



fe-1 



s=l 



<r m A - 2m (i-p) fe - ( 



-fe 

2/Tt \ _ 2 



(l-p)c 



-1 ' 

2m\ _ ^ 



(32) 



with T m = M(iV - -L)- 2m (2cr /2 ) m < oo. It follows that if 



m is taken such that 



(i- P K 2m >i, 

then liniA„-i.oo Gfc(A ) = 0, and for all k € N 



(33) 



e fe <(i- P ) fc -M i + r m A^ 2m — 



i 



□ 



Lemma 5.3: The discrete probability measure P(T z +i 
T z — k\ x-fz , A7; ) satisfies 

P(T Z+1 -% > k\x % ,A T J > (l- P ) k -\ 
for all realizations of x 7; , A 7; . 
Proof: This follows since 



□ 



fe-i 



fe-i 



< MS" m . 



(28) 



Now 6>i = 1 by definition, and for k > 1, 

e fc < e k - 1 (l~p) + Ce- 3k . (29) 

We obtain, 

fe-i 

e fc < e 1 (i-p) fc - 1 + ^(i-p) fc -^ 1 c e - H = 

s=l 

fe-1 



□ 

As a consequence of Lemma 15.21 and Lemma 15.31 the 
probability P{T z+ i-T z = k \ x%,Ar m ) tends to {^-p) k ^ 1 P 
as A7; — > 00. □ 



2) Proof of Proposition \3.2\ Let the values taken by 
log 2 (F(-))/s be {-1,0, B}. Let 



L 



z ,A,B 



{n e N,n > \og 2 {L')/s 



< 9i(l - p)^ 1 + J] M(l - p) k - s - 1 E- r " 

s=l 



(30) 



: 3N A € N, 7Vg e N, n = -iVjl + A^B + z }- 
Since we have by ( TTOb 

At+i = H(A t , \h t \,T t )A t , 
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it follows that, 

log 2 (A t+1 )/a = log 2 (ff(A t , \h t \, T t ))/s + log 2 (A t )/s 



is also an integer. We will establish that L, Zg ^ ^ forms a 
communication class, where zq = log 2 (Ao)/s is the initial 
condition of the parameter for the quantizer. Furthermore, 
since the source process x t is "Lebesgue-irreducible" (for 
the system noise admits a probability density function that 
is positive everywhere) and there is a uniform lower bound 
L' on bin-sizes, the error process takes values in any of the 
admissible quantizer bins with non-zero probability. In view of 
these, we now establish that the Markov chain is irreducible. 

Given /, k e L 2q there exist N^,N^ e N such that 
I — k = -N^A + NgB. In particular, if at time 0, the 
quantizer is perfectly zoomed and A = 2 sk , then there 
exists a sequence of events consisting of Ng erasure events 
(simultaneously satisfying \h t \ < 1) and consequently iVj 
zoom-in events taking place with probability at least: 



pP(d t £ [~(a2 R ' - \a\)L'/2, (a2 R ' - \a\)L'/2]) 



x((l~p)P(\d t \<62 R '- 1 L') 



>o, 



(35) 



so that P(A Na+Ns = 2 sZ |A = 2 ks ,x ) > 0, uniformly in 
xq. In the following we will consider this sequence of events. 

Now, for some distribution JC on positive integers, £cl 
and A an admissible bin size, 



K(n)P[(x n ,A n )e(Ex{A}) 



x ,A 



>K AoA iP(E,A) 



Here i^Ao-A, denoting a lower bound on the probability of 
visiting A from Ao in some finite time, is non-zero by (l35T l and 
ip is positive as the following argument shows: Let t > be 
the time stage for which A f = A and thus by the construction 



in 



with I hi 



< 1: |axf_ 



bu 



t-l 



< 



(\a\/a) — . Thus, it follows that, for Ai,Bi E 



|a|A t _i/2 
., Ai < Bi, 



P art € [Ai.Bi] \ax t -i + bu t -i\ < |a|A t _i/2, A t _ a 



F 



> min 

Nl<# (|a|/a) 



= P( axt-i +bu t -i +d t -i E 

\ax t -i + bu t -i\ < |a|A t _i/2, A t _i 

P(rf 4 _i e[A 1 -z,B 1 -z]j >0 (36) 

Now, define the finite set C' A := {A : V < \A\ < 
E N}. The chain satisfies the recurrence property 
that P(x,a)(tCxxC' & < °°) = 1 f° r an Y admissible (x,A). 
This follows, as before in Section lV-B.ll from the construction 
of 

e fc (A,a!) := P(Ti >fc|aj,A), 



where 



Ti = inf(fc > : |arjfe| < 2 R '~ 1 A k , x = x,A = A) 



and observing that Qk{A,x) is majorized by a geometric 
measure with similar steps as in Section IV-B.ll Once a state 
which is perfectly zoomed, that is which satisfies \xt\ < 
2 R _1 A f , is visited, the stopping time analysis can be used 
to verify that from any initial condition the recurrent set is 
visited in finite time with probability 1. 

In view of d35l). we have that the chain is irreducible. 



We can now show that the set C x x C' A is small. We will 
show first that this set is petite: A set D e B(X) is petite if 
there is a probability measure JC on the non-negative integers 
N, and a positive measure \i v satisfying /i(X) > and 

oo 

Y P n (x,E)JC(n) > H P {E),for all x G D, and£ E B(X). 

n=0 

By Theorem 5.5.7 of [27], under aperiodicity and irreducibil- 
ity, every petite set is small. To this end, we will establish 
aperiodicity at the end of the proof. 

To establish the petite set property, we will follow an 
approach taken by Tweedie [38] which considers the following 
test, which only depends on the one-stage transition kernel of 
a Markov chain: If a set S is such that, the following uniform 
countable additivity condition 



lim sup P(x, B n ) = 0, 



1 xes 



(37) 



is satisfied for every sequence B n I 0, and if the Markov chain 
is irreducible, then S is petite (see Lemma 4 of Tweedie [38] 
and Proposition 5.5.5(iii) of Meyn-Tweedie [27]). 

Now, the set C x x C' A satisfies d37l i. since for any given 
bin size A' in the countable space constructed above, we have 
that 



lim sup P (x f+ i,A f+1 ) E (B„ x A') 

n ^°°(x,A)eC x xC A ' 



x t = x,A t = A 



lim sup P [ (ax + bu t + d t , A t +i) 

™^°° (x,A)eC x xC' A ' 



E (B n x A') 



x t — x, A t = A 



= lim sup P (d t , A t+ i) 
E ((B„ - (ax + bu t )) x A'^j 



x t = x, A t — A 



0. 



This follows from the fact that the Gaussian random variable 
d\ satisfies 

lim sup P(di E A n ) = 0, 

uniformly over a compact set Co, for any sequence A n I 0, 
since a Gaussian measure admits a uniformly bounded density 
function. 

Therefore, C x x C' A is petite. 

If the integers A, B are relatively prime, then by Bezout's 
Lemma (see [1]), the communication class will include the 
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bin sizes whose logarithms are integer multiples of a constant Consequently, under the bound ©, there exist 6 > 0, &i < 

except those leading to A < L'. oc,F' > such that, 

We finally show that the Markov chain is aperiodic. This E\V ( A II A 1 

follows from the fact that the smallest admissible state for the L (H 7i+u T z+ i)\ x T z , TA 

quantizer, A* = L', can be visited in subsequent time stages — ^o( x 7;>ArJ — bo + &il{|A Tz |<F'}- (42) 

with non-zero probability, since This combined with Proposition EU eqn. gD, and dHJ), 



min P(dt £ [-2 R '" 1 A*-aa;,2- K '" :L A*-aa;l) )p> 0. 

\x\<A*/2 ' 



Corollary 12. 1 1 leads to positive Harris recurrence. □ 
4) Proof of Theorem \3.2\ First, let us note that by ( f30b and 
3) Proof of Theorem \3J} With the Lyapunov function <GD. for every k > 0, we can find A sufficiently large such 
V (x t ,A t ) = log(Af) + B , for A r , > L, we have that 

E[Vb(a; r=+1 ,A r , +1 ) | x Tm ,A Tm ] 
= B + P(T z+ i -% = l\x T ., A r J 



that 



lim ^ ^ = 0. 



xf21og(a)+log(A2-; 

oo 

+ £ log(A^ +fe )P(r z+1 -T s = k\x T ., A r J 



fe=2 

Thus, the drift satisfies: 



t-K» (1 - p + k)* -1 

Under the assumed bound (1 — p)|a| 2 < 1, we can fix k > 
such that (1 - p + n)\a + S\ 2 < 1. 

Next, observe that for all initial conditions for which \ho\ < 

1, 

E[V 2 (x Tl7 A Tl ) | x ,A ] 



lim ■ 

An— >oo 



E[V (xt +1 ,A Tz+1 ) | x % , A r J - V (x Tz , A r J 

OO 

= £ 2 log((|o| + ^^^^P^+i - T z = k | zr., A r J 



= lim 

A q—^oo 



V 2 (x ,A ) 
E[A 2 Ti \x ,A ] 
A 2 



A lim ^2j2 P ^= k ) E l A k\^ = k ' x o,A ] 



fc=l 
oo 



fc=l 



= 2^T(fc - 1) log(|o| + 5)P(%+i -% = k\ x % , A r J 



fe=i 



lim a 2 V P(7i =fc)(|a|+(5) 2(fc - 1) 

A n — >oo L — ' 

fc=l 

i 



(43) 



+21og(a). (38) 

= put 

By (|3D, the summability of J2T=i G k {A Tz ), and the domi- 1 ~ ~ pXM + 5 ) 2 ' 

nated convergence theorem, where the last equality follows from Lemma 15.21 and the 

oo dominated convergence theorem, 

lim V(fc - 1)((1 -p) k ^ 1 + Gfe(A r J - (1 -p) k ) Now ' if 03 holds ' we can find a such that R> > 

At-, — foo ^ — ' 



k=l 



lo g2(l a l/a)' and 



£ lim (k - 1)((1 - p)^ 1 + G fe (A T J - (1 - P) fe ) 

' — ' At, — >oo 



pa 



k=l 



< 1, 



(44) 



fe=i 



Provided (O holds, it follows from Lemma |BT2l and Lemma 
that for some bn > 0, 



l-(l-p)(\a\ + S) 2 

and simultaneously (O is satisfied. We note that d44b implies 
(39) (|9]| since by Jensen's inequality: 

log(pa 2 +(l-p)(\a\+5) 2 ) >plog(a 2 )+(l-p)log((| a |+<5) 2 ), 

and (O is equivalent to the term on the right hand side being 



ArSoo { E ^°( a:r -+ 1 ' A T+i) I ST., A r „] - U (a;r , A T J} | Q establish the required drift equation, we first establish 

the following bound for all z > 0: 

«E["f; ^I^Aol^A 2 ,^ 2 ^'- 1 ), (45) 

m=Tz 

< — &o ■ (40) for some k > 0. 

-«— i a . . , , , jjur/j£j Without loss of generality take z — so that T = 0. 

For At in a compact set and lower bounded by L defined b 1 z 

by CD, E[log(A 2 - +i ) | x T . , A r J is uniformly bounded. This 0bserve ****** X > 0, by Holder's inequality, 
follows from the representation of the drift given in 
Finally, since, 



21og(a)+2 lim { V(fe - 1) logflal + 8) 

At,->oo * — ' 
k=l 

xP(T 2+ i -%=k \ xr,A r J} 



71-1 



E[£ x 2 t \x ,A } = E[Y,l{t< ri }Xt\xo,A } 



t=o 



t=0 



G fc (A T J < (l- P )^r m A-- i _ ((i _^ 2m) _ i ; 
it follows that J2T=i Cfc(Ar )fc < oo and as a result 



< 



i+x 



•to, A ] 



sup E xr ^ A T, [7i+i - 7^] < oo. 

XT 2 ' A Tz 



(41) 



X E[; 



2 (^) 



»o, A " 



(46) 
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Moreover, for some £>2 < oo, 

21 1+ x 1 

E[x t ( * >o,Ao] 

= E[a 2 ^) (x + £ a— ^) 2 ^) 1*0, A ] 

t=0 

= |a| 2 ^>E[(x + |] a-^d,)) 2 ^ Iso, A ] 

i=0 
00 

< lal^^E^o + ^ a- 4 - 1 ^) 2 ^ l^o, A ] 



i=0 

xE[( 



• TO + Er - a ^ ld -) 2 ^|xo,A ] 



2«'-iA 

| a |2*(^) (2 «'-l Ao) 2^ 

xEpo+ ^g! 1Ao * ) a ^l«o,Ao] 

i+x , ,0*/i+2O 



< J B 2 (2«'- 1 A ) 2 ^|a| 2t ^), 



(47) 



where the last inequality follows since for every fixed |/io| < 1, 
the random variable ho + (Y^Lo a~ t ^ 1 di)/(2 _1 Ao) has a 
Gaussian distribution with finite moments, uniform on Ao > 
L'. 
Thus, 

71-1 

E[£ *?ko,A ] 

4=0 



4=0 



<^T(E[(l {t< T l} ) l +x\x ,Ao. 



x(S 2 (2«'- 1 A ) 2 ^|a| 2 ^) N " 1 " 



4=0 



(2«'- 1 A ) 2 ^ P(7i >i+l|x ,A ; 



B2\a\ v f ; 



<Cs 2 (2 fl/ - 1 A ) 2 

for some (^b 2 < 00. 

The last inequality is due to the fact there exists k > such 
that 

P(7i >i|x ,A ) 

lim — . , : = 0, 

4-s-oo (l-p + K)*- 1 

and we can pick x > with (1 — p + 4t)|a| 2( - 1+x ^ < 1. Such 
X and k exist by the hypothesis that (1 — p)|a| 2 < 1. 
Hence, with < e < 1 - pa 2 /[I - (1 - p)(|a| + <5) 2 ], 

5(x,A)=eA 2 , f(x,A) = 



Cs 2 2 2 ( ff -D 



C a compact set, and V^x, A) = A 2 , Theorem 12. 1 1 applies 

and lirrif-j-oo E[x 2 ] < 00. □ 



5) Proof of Theorem li.il The proof follows closely that of 
Theorem 13.21 

Again by Holder's inequality, for any \ > 0, 

-71-1 



E 

4=0 



Z4 



xo, A 



S 1 {*<7i}l a! *| r 



4=0 



*o, A 



4=0 



<^ E[(l {t<Tl} )^|x ,A ] 



x( E[|x t r^|x ,A 



(48) 



As in (07]i, for some B rn < oo, 

E[|x t r^)|x , A ] < 5 m (A ™2™(«'- 1 )) 1 ?|ar t ( i ?) 
and consequently, 

ri-l 

E[]T |xt| m |a:o,Ao] 
4=0 



£(P(Ti>t + l|a:o,Ao)) [Bl 



i+x / x 

+x| a im4 



<c Sm (2 fl '- i A r 



(49) 



where, once again, the last inequality is due to the fact that 
there exists a k > such that 

P(7i >t|x ,A ) 
lim — — : = 0, 

and we can pick \ > such that (1 — p + K)|a| m ( 1+X ) < 1; 
such x an d ft exist by the property (1 — p)|a| m < 1, 
Hence, with < e < 1 - pa rn /[l - (1 - p)(|a| + <5) m ], 

S(x,A)=eA m , f(x,A) = - J (R ,_ 1} \x\ m , 



C a compact set, and V m (x,A) = A m , Theorem 12. 1 1 applies, 
establishing in the desired conclusions, and in particular that 

lim^oo i?[|a;t| m ] exists and is finite. □ 
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